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Introduction 


Higher education experienced an unprecedented and unplanned shift to remote instruction in response 
to the COVID-19 pandemic in spring 2020. For the first time, the great majority of postsecondary faculty 
and students were engaged in remote instruction. Faculty members who had never taught (or learned 
through) online courses were challenged to teach in a new format. Students who had never learned 
online before, and were not expecting to do so, had to shift to learning at a distance at the same time 
they were coping with abrupt changes in their living circumstances. 


With support from a National Science Foundation RAPID grant, we set out to study the nature of 
teaching and learning in STEM courses during this period, with a focus on the experiences of students of 
color and those from low-income backgrounds. These students are more likely to have outdated 
computing devices and limited internet connectivity that would restrict their ability to participate online. 
In addition, students with disabilities and those who struggle with academic English, have weak math 
backgrounds, or lack confidence about their ability to learn STEM subjects could be especially vulnerable 
to deficiencies in online course materials and to the loss of hands-on activities and face-to-face support. 


The abrupt transition to remote instruction in response to COVID posed significant challenges for both 
students and instructors. This report provides data on the prevalence of the different kinds of challenges 
students faced and the ways in which low-income and minoritized students were differentially affected 
by the shift to remote instruction. It also describes the nature of STEM courses in spring 2020 as they 
were taught at a distance from the perspectives of both students and instructors. These descriptions are 
complemented by survey data on the prevalence of online instructional practices that are generally 
recommended in the online learning literature and analyses of the relationship of these practices to 
student satisfaction with their STEM course. 


As the pandemic has persisted, restrictions on classroom-based instruction have extended much longer 
than we envisioned in spring 2020. Colleges and universities are looking for ways to design courses that 
are resilient in the face of epidemics and other interruptions to campus-based teaching while also being 
equitable to all students. This report highlights challenges and equity issues that surfaced in spring 2020 
and that should be considered in course design and implementation, even beyond the pandemic. Its 
description of specific practices associated with higher student satisfaction with their STEM courses has 
applicability far beyond the first semester of pandemic-induced remote learning. 


This report describes findings from two research activities conducted concurrently: 


e Survey of a nationally representative sample of over 650 undergraduates who were taking STEM 
courses that included in-person meetings when they began and had to switch to entirely remote 
instruction 


e Qualitative descriptions of 28 STEM courses offered by nine institutions, based on interviews 
and focus groups with students and instructors. 


The full set of student survey results have been reported elsewhere (Means & Neisler, 2020).? In this 
report we provide selected survey findings along with related descriptions of STEM remote teaching 
practices and learning experiences from the course case studies. 


thttps://digitalpromise.dspacedirect.org/bitstream/handle/20.500.12265/102/NSF%20COVID FINAL Aug13.pdf?s 
equence=3 
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Taken together, the survey and case studies provide complementary findings with respect to four 
research questions: 


e What was the nature of online learning in STEM courses during the COVID-19 pandemic? 


e What challenges did students experience in learning online and how did those challenges vary 
for different kinds of learners? 


e What instructional practices were associated with greater student satisfaction with their STEM 
courses during the COVID-19 pandemic? 


e How did prior use of digital courseware in blended courses using adaptive courseware before 
COVID affect the transition to remote instruction? 


Methodology 


Student Survey 


Sample 
The Survey of Student Perceptions of Remote Teaching and Learning, developed by Digital Promise in 
collaboration with Langer Research Associates, was administered to a national sample of U.S. 
undergraduates, age 18 and older, who were taking college courses for credit that included in-person 
class sessions when the COVID-19 pandemic hit and had to finish the course by learning at a distance. 
Ipsos Public Affairs conducted sampling and data collection for the survey via its online, probability- 
based KnowledgePanel,® which administers internet-based surveys to randomly recruited participants in 
U.S. households. 


Of the 1,709 panelists initially responding, 1,014 were qualified undergraduates taking for-credit college 
courses that started with in-person classes and later went fully online in spring 2020. The median survey 
completion time was 10 minutes. Quality control flagged respondents who completed the survey in less 
than one-third of the median time, leading to the removal of six cases, for a final sample size of 1,008. 
Sixty-two percent of the national survey sample of 1,008 had been taking one or more STEM courses 
with in-class meetings that had to go online during the COVID-19 pandemic, and this report describes 
the responses of these 620 students. 


Method 
After answering screening questions designed to ascertain eligibility for the survey, respondents were 
asked to think about one particular course as they responded to the remaining survey questions. The 
survey asked respondents to answer the survey questions for a STEM course if they were taking one. 
Students who were taking more than one STEM course were asked to respond for the one they thought 
was most important for their future goals. (Similarly, students who were not taking any STEM courses 
were asked to respond for the non-STEM course they considered most important for their future goals, 
but their responses are not included in this report except where explicitly noted.) 


Field work was conducted between May 13 and June 1, 2020. Starting May 15, email reminders were 
sent to non-responders every two days; those who did not respond by May 29 also received a reminder 
by telephone, and incentives (panel “points” that can be redeemed for prizes in future sweepstakes) 
were increased, first on May 27 and then again on May 29. 


Students self-identified in terms of race/ethnicity when they joined the lpsos survey panel. For survey 
data analysis, response codes to items regarding race and ethnicity were used to form four groups: Black 
Non-Hispanic, Hispanic, White Non-Hispanic, and Multiracial or Other Non-Hispanic. Analyses of survey 
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responses compared the experiences of (non-Hispanic) White students to those of Hispanic and Black 
(non-Hispanic) students (referred to in aggregate as “minoritized student groups”) to better understand 
the disparate impacts of the transition to remote instruction. (There were too few students in other 
race/ethnicity categories to support reliable statistical estimates.) 


Data were weighted via iterative proportional fitting to benchmark distributions of adult undergraduate 
students with less than a bachelor’s degree from the 2018 American Community Survey. Weights were 
trimmed at 0.23% and 99.77% and then rescaled to match the sample size of all respondents. The final 
weights for qualified respondents were extracted from the above weights and re-scaled to sum to the 
sample size of qualified respondents. The average design effect for the final weight is 1.35, for a margin 
of error of 3.6 percentage points for the full sample and 4.6 points among students who took a STEM 
course. Error margins are larger for subgroups. 


Weighted proportions of responses were used to assess statistical significance via Chi-square tests, 
where p < 0.05 was used as the significance criterion. Data exploration and statistical analysis was 
completed in R Studio version 1.3.959 (R Studio Team, 2020). 


Course Case Studies 


Sample 
A convenience sample of nine broad-access colleges and universities that had been working to redesign 
their gateway STEM courses to include adaptive learning courseware was recruited for the qualitative 
research. (Adaptive learning courseware, such as ALEKS, LearnSmart, and MyMathLab, organizes course 
content by learning objective and uses assessments of student performance on an ongoing basis to 
determine an individualized path through the course content.) Each of these institutions identified one 
or more STEM courses that had undergone redesign to include use of adaptive courseware. Institutional 
project leads invited instructors of course sections using adaptive courseware and any instructors of the 
same courses that had not been using adaptive courseware to be part of the COVID teaching and 
learning research. Willing instructors were asked to send an email informing their students of the 
opportunity to participate in a focus group or interview between April and June 2020. The email invited 
students to indicate their interest by filling out a web form maintained by Digital Promise. Digital 
Promise researchers then contacted students directly to schedule participation. Instructors were not 
informed of which of their students had agreed to participate in the focus group. There were no criteria 
for participation other than having been enrolled in a course that implemented adaptive courseware in 
at least one course section that semester. A limitation of this recruitment approach is that students who 
volunteered were not necessarily representative of all students in these courses: Those students most 
severely impacted by COVID and most disconnected from their courses may not have even seen the 
invitation to participate. 


Ninety-two students taking STEM courses at nine postsecondary institutions participated in an interview 
or focus group. Of these, 73 students were enrolled in a 4-year institution and 19 students were enrolled 
in a 2-year institution. Eighty-one of these students (88%) were in sections of a STEM course that had 
been using adaptive learning software prior to the COVID-19 outbreak. In most of these courses, 
students’ work with the courseware counted for 30% or more of their course learning time, which 
meant that the course satisfied customary criteria for being a blended course (Allen, Seaman, & Garrett, 
2007). The STEM courses that students in case study courses were taking were in the fields of biology, 
chemistry, geology, mathematics, and statistics. 


In addition, each instructor with four or more students participating in focus groups was interviewed by 
a member of the research team. In all, 28 instructors from the same nine institutions participated in an 
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interview about how they conducted their STEM course before and after the shift to completely remote 
instruction. The great majority of these (89%) had been teaching with adaptive courseware. 


Method 
Depending on students’ schedules, they were either interviewed individually or participated in a focus 
group with up to four other students. In both interviews and focus groups, students were asked to 
describe different aspects of their course experiences pre- and post-COVID. Interview questions asked 
these students how they used the courseware (if applicable) and how much time they spent using it 
before COVID as well as afterward. Students in course sections that were not using adaptive learning 
software were asked about any digital learning tools that were used in their course. Students were 
asked about the computing device they typically used when working on the course and any technology 
issues they may have experienced. 


Questions about the course post-COVID included probes for how they accessed the internet, any 
changes to the computing device they used for the course, and any technology issues they experienced 
after instruction shifted online. We asked students how the structure and organization of their course 
changed once it was fully online (e.g., expectations for course participation, changes to course 
assignments or grading), for their perceptions of course difficulty, and what they missed about the 
course as it was taught before COVID. Students were also asked about challenges they encountered with 
continuing their learning at a distance (such as finding a quiet place to work) and how prepared they felt 
they were to take subsequent courses in the same subject area. Finally, students were asked about their 
overall satisfaction level with the course post-COVID and, in the case of students in sections using 
adaptive learning systems, if they felt the use of the courseware before the coronavirus outbreak helped 
them continue their learning in the course after it was fully online. 


In their interviews, instructors were asked about the role of the adaptive learning system or other digital 
learning tools in their course, their instructional practices pre- and post-COVID, the biggest challenges 
they and their students encountered when transitioning to remote learning, and institutional supports 
the instructor received for transitioning the course. Instructors were also asked to reflect on how well 
their students had learned in the course and how well prepared their spring 2020 students were for 
future STEM courses. 


Findings 


Challenges to Continuing Courses Post-COVID 


Technology access 
When STEM courses moved unexpectedly to remote instruction, students no longer had access to 
internet connectivity and devices provided on college campuses. The survey data provide an indication 
of how widespread technology access issues were during the first semester of the COVID pandemic. 
Internet connectivity issues were serious enough to interfere with students’ ability to attend or 
participate in their STEM courses at least occasionally for 46% of STEM students, with 15% of students 
reporting that they experienced such problems often or very often. 


Almost a quarter of students taking STEM courses (23%) reported experiencing hardware or software 
problems serious enough to impact their ability to attend or participate in their STEM course at least 
occasionally, with 7% reporting that they had such problems often or very often. Hardware and software 
problems tended to be experienced by the same students who had difficulties with their internet 
connection. 
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Considering the two kinds of technology issues together, about one in six students taking a STEM 
course had internet connectivity and/or hardware/ software problems often or very often. 


Reflecting long-standing inequities, technology access issues were more prevalent among students of 
color and those from lower-income households. Survey responses revealed that students from 
minoritized race/ethnicity groups were more likely to experience internet connectivity problems often 
or very often (20%) compared to students from non-minoritized groups (12%). Students from lower- 
income households (annual incomes under $50,000) were also twice as likely to experience internet 
connectivity problems often or very often (22%) compared to students from households earning 
$100,000 or more (11%). 


Although major hardware and software problems were less frequent than major connectivity problems, 
their distribution across different student groups followed the same pattern, further exacerbating 
inequities. Larger proportions of Hispanic and Black students (11% combined) experienced hardware or 
software problems often or very often compared to just 5% of White students. Frequency of serious 
hardware and software issues varied also by household income. Just 4% of STEM students from high- 
income households reported having these problems often or very often compared to 10% of students 
from lower-income households and 8% of students from middle-income households (annual income of 
$50,000-$100,000). 


Student comments during interviews and focus groups give texture to these numbers. One student in a 
case study chemistry course explained, “I've definitely had some issues with internet connection. | live 

pretty close to the mountains, so | get like one bar of service at all times. | can't even make phone calls 
from my house, so | barely get any service.” 


Another student described having to work in a less-than-ideal place to get internet access. 


I’ve been staying above my garage in my house and so internet access for me only works in my 
bathroom. | have to do a lot of my work like studying and watching the videos in my bathroom, 
which is inconvenient. 


In addition to the direct effects of limited technology access on course participation, there were also 
indirect effects associated with fear of losing connectivity. A chemistry student described heightened 
anxiety around course examinations, 


With the unstable internet that | have, | go in with more anxiety than | would normally if | were 
in person because | don't know whether my internet is going to go out or I'll have to contact 
someone to get the test reset, and then see if they're going to let me retake my test. 


Instructors in the interview sample were aware of the kinds of technology issues faced by some of their 
students. A statistics instructor, for example, shared the experience of a student who was disadvantaged 
because of limited internet access. 


| had a student who was doing fine before COVID. After COVID, | don't remember exactly how 
many assignments he missed but | emailed and said, “Hey, | noticed that you haven't completed 
this assignment. Is there anything | can doP” He said, “Well, we're struggling with the internet 
here. | don't know what to do to complete them.” | said, “I can extend the deadlines and you can 
even drive to the school parking lot. If you have a laptop, then you can use the campus Wi-Fi, or 
even use the Wi-Fi at a McDonald's inside your car or something.” Again, these are things that if 
a student doesn't have a laptop, then they can't do that. 


Instructor awareness of their students’ technology issues influenced the way they conducted their 
courses. In addition to extending deadlines (as the statistics professor above did), instructors often 
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made their course materials available asynchronously, citing students’ technology access issues as the 
rationale. 


| realized that, of course, not all students have the hardware for Zoom. Phones are pretty 
ubiquitous and even computers are fairly ubiquitous, but Wi-Fi is not. A lot of students really rely 
on public spaces in the university for Wi-Fi. Wi-Fi is not ubiquitous. Having a working mic and 
camera is not ubiquitous, or even speakers, is not necessarily ubiquitous. Understanding that, | 
would offer the classes via Zoom, but | would also post the recordings online. That way, if a 
student couldn't access a live Zoom recording because their hours at work changed, or they don't 
have access to Wi-Fi, or whatever it may be, they could at some point stream that later. There 
was this synchronous lecture, but it could also be accessed asynchronously. 


Non-technology challenges 
After answering questions about their experiences with technology after instruction moved online, 
students completing the survey were given a list of other potential challenges to continuing their 
learning remotely and asked to indicate the extent to which each was a problem for them. Figure 1 
shows survey responses for the six challenges included in the survey. 


Figure 1. Experience of problems participating in their STEM course at a distance 


Staying motivated to do well 
in the course 


Finding a quiet place where you 
could do the course online 


Not knowing where to get help 


with the course Major Problem 


ai 
B Minor Problem 
i 


Not a Problem 


Fitting the course in with your 
home/family responsibilities 


Feeling too unwell, physically or 
emotionally, to participate 


Fitting the course in with your 
work schedule, if you work for pay 


0 25 50 75 100 
Percent of Students 


As illustrated in the figure, staying motivated to do well in the course was the most prevalent 
challenge to students’ completing their STEM courses at a distance. The proportion of students who 
reported maintaining motivation as a major problem (45%) on the survey was double that for the next 
most frequently experienced challenge (finding a quiet place to work on the course, a major problem for 
21%). Student comments in focus groups and interviews illustrated the nature of the challenges 
included in the survey. 


Maintaining motivation and focus. Many students reported difficulties with staying motivated, focused, 
or “connected” to course material after their courses transitioned online in response to COVID. 
According to one math student, the most difficult part of the remote learning experience was listening 
to the lectures. “I would usually just zone out,” the student said. Another student described a similar 
challenge in their chemistry course, 

Well, for me one of the struggles has been focusing since you're staying in your own home so you 

get distracted easily and often enough. Also, at times just really connecting with the course. 
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Before in person, you could connect with the course more than not being there, so that was one 
of the struggles | had. 


Many students described the temptation to put off watching lectures when there were no scheduled 
class meetings. One biology student explained, “What | miss is just the whole schedule and routine thing 
of going to class, kind of like some mandatory thing you have to do.” Another biology student explained, 


With meeting face-to-face, you're holding yourself accountable because you had to go to class to 
know what was going on. But when everything went online, it was easy to procrastinate and put 
off watching the lectures. | know | personally was stuck with watching, | think | had seven or 
eight lectures that | had to watch over the weekend before the final just because | found myself 
not really wanting to wake-up early and watch them. 


Finding a quiet place. After the shift to remote instruction, many students found themselves working in 
an environment that was not conducive to learning. Students reported being distracted by elements of 
their environment when working on their course, often because of other people in the home. “Finding a 
quiet place was the worst thing ever,” said a math student. “There's dogs here. | live with my boyfriend. 
It's chaotic here. Finding a quiet place was definitely one of the biggest struggles.” A chemistry student 
felt similarly, “I get really distracted very easily,” said the student. “Having online lectures while my 
parents are stomping around upstairs and my dog's barking, | can't focus well at all.” Another chemistry 
student described being distracted by their younger sibling, 


My room is right beside my little brother's playroom. He's always on his Xbox screaming, and of 
course, my room is not as soundproof as much as | wish it was. The noise—I’d be trying to take a 
test or studying for a test and all of a sudden, a little sibling walks into my room and just 
completely throws off my train of thought...So, I'm usually downstairs outside on the porch trying 
to find a quiet place to study. But of course, it's Mississippi and it's starting to get quite warm, so 
that's not going to work for much longer. 


Students were particularly distressed by lack of a quiet place when taking course exams. A chemistry 
instructor shared some of the situational challenges that students experienced when taking their exam 
at home. 


During the exam | had one student who emailed me and she's like, “I'm really sorry, but you're 
going to see that | was talking during my exam.” She's like, “I had warned my brother that | was 
taking an exam and he was screaming in the background and | couldn't concentrate, and finally, 
| just got fed up and | snapped.” | had another student who emailed beforehand and said, “Look, 
| told my roommate, | have an exam, but both of our computers are in the same room. He 
doesn't seem that concerned that | have an exam, so if there's somebody talking in the back, I'm 
sorry. | tried.” He was actually one that we had to live Proctor and my TA [teaching assistant] 
emailed me and she's like, “Somebody screaming in the background.” His friend was playing an 
online game and was just screaming and there's nothing they could do. They had to work 
through it. | have so many more examples of things like that, where | think that was a huge 
struggle. 


Not knowing where to get help with the course. When STEM courses switched to remote instruction, 
many students had to use learning and communication technologies that were new to them and 
encountered changes in course requirements and deadlines at the same time they lost in-person contact 
with instructors, peers, and teaching assistants. Although many colleges continued supplemental 
instruction (SI) and tutoring services online, students did not necessarily know how to access these 
resources from a distance. Almost 1 in 5 survey respondents cited not knowing where to get help with 
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the course as a major problem. Students in case study courses described their feelings of needing help 
they couldn’t access. 


It was just really frustrating, | guess, to just [try to] understand stuff and not really know how to 
get the help you needed when you aren't seeing your professor or seeing your classmates or 
anything like that.” 


| think the hardest part is not being able to get that extra help on the topic, like going to SI 
[supplemental instruction] sessions or tutoring. | think that's just the hardest part about online 
classes. You could go online during a certain hour and have a Zoom meeting, but | didn't think it 
was as useful as an actual face-to-face one. | tried that way and for tutoring, but the meetings 
were always filling up quickly, so it was hard, you had to compete to get attention. 


Competing family responsibilities. Conflicts with family responsibilities were deemed a major problem 
by almost 1 in 5 survey respondents. Students described the conflict between doing coursework and the 
extra layer of responsibilities for children or other family members at home. 


It's a totally different environment, going from leaving the college to now being back with my 
parents and stuff like that. There's a lot more like needs for me at home, helping take care of my 
sister and helping to do chores and my parents want it done right then and right there. When | 
was at school, | knew exactly how | could plan for myself when | wanted to do things and how | 
wanted to do things. 


| would get really distracted by my family... | live with my grandparents and my great- 
grandparents. | would always have to help them out. | always have to cook for them, take care of 
them. Even simple things, like turning on the TV so they can watch the news. I'm African...my 
African parents, | always have to obey them, | can't tell them, ‘No’, that is disrespectful. | always 
have to make sure | do what they want. 


| have a two-year-old and we share the room, so when she takes her naps, that's the only time | 
can really study. Right now, she's next to me, I'm waiting for her to wake up. When ! was on 
campus she was at daycare and | had a chance to just sit and focus just on school instead of me 
worrying if she's okay. 


Feeling too unwell. Roughly 1 in 7 survey respondents rated feeling unwell, physically or mentally, as a 
major problem when they were trying to complete their STEM course online. In the words of several 
case study students, 


Probably, dealing with the different quarantine things, different regulations that they put in 
place: the six feet, the masks, and all of that. That really put me into a trouble because I'm not a 
good change person. | don't deal with change very well. It was all sudden, and they told us, “Stay 
inside.” 


With everything that's going on, and going to the grocery store is dangerous now. It takes so 
much out of my day just to go run errands or something like that. If | have to leave the house, | 
have to wear protective gear and stuff like that. | have so much going on besides class... The 
main thing for me right now is survival and staying healthy, but then, | have like all this difficult 
work that makes it more stressful for me, and just the environment and life is different, and 
school is still hard, if not harder. 


Conflicts with work. The proportion of survey respondents who cited conflict with work schedules as a 
major challenge to completing their course was relatively small (8%), but the impact on those particular 
students could be large. As a case study student in a chemistry class described, 
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My schoolwork, | usually end up doing late nights or over nights on my days off... I’m at work 
from 8:00 AM to 7:00 PM, four days a week. That takes away a good chunk of classes that | 
could probably take and then | have a child, so that takes away a good amount of time. | just 
really do make sure | calendar in when everything is due and then plan from there about what 
time block after | get off of work or overnight or on my days off that | can set aside for studying, 
for each class and getting work completed on time. 


As described in Means and Neisler (2020), the challenges posed by COVID were experienced most 
acutely by students of color, females, and students from low-income backgrounds. Hispanic students 
reported higher levels of impact for six out of the seven challenges covered in the survey, including 
fitting coursework into family and work responsibilities, finding a quiet place to work, knowing where to 
go for help, and feeling too unwell physically and emotionally to focus on coursework. They were also 
most likely to have problems with internet access, together with students from families with annual 
incomes below $50,000. Recent reports showing inequities in cases, hospitalizations, and fatalities due 
to COVID help to explain why minoritized students, especially Black and Hispanic students, were most 
likely to cite their physical and mental health as major challenges to participation in coursework.” As 
found in previous research (Wladis, Hachey, & Conway, 2018), female students, especially women of 
color, were more likely than others to have experienced disruption due to loss of childcare. 


How Instructors Responded to the Challenges of Remote Teaching and Learning 


Instructors had very little time—sometimes just a day or two—to prepare for shifting their course online 
in the spring of 2020. Particularly those instructors who had never taught online before found 
themselves making it up as they went along. The course case studies provide examples of the kinds of 
accommodations instructors were able to make on short notice. 


Use of shorter instructional segments 


Guidelines for designing online courses recommend keeping instructional segments, including videos, 
short (Scagnoli, 2012). Some of the instructors of case study courses followed this advice in spring 2020 
when they moved their courses to remote delivery. One geology instructor, for example, broke each 
course lecture up into two to three shorter videos that were 10-25 minutes long. He inserted blank 
screens within each video containing digital notes on what would have been written on the class 
blackboard. A mathematics instructor using an asynchronous course format went a step farther, 
creating short instructional videos with answers for some of the questions that students had about the 
statistics software they were using. 


Other case study course faculty chose to create videos that resembled the in-class experience, in some 
cases showing themselves lecturing at a podium in an empty classroom or recording a set of PowerPoint 
slides with an inset of their face as they gave the lecture associated with the slides. While this approach 
provided an element of continuity with the course experience pre-COVID, it did mean that students 
were viewing long lectures online. 


A number of students in interviews and focus groups commented on the length of recorded course 
lecture material as an impediment to maintaining focus on course content. A biology student taking a 
course where the instructor recorded longer video lectures described feeling less engaged with the 
material, 


? https://www.cdc.gov/coronavirus/2019-ncov/covid-data/investigations-discovery/hospitalization-death-by-race- 
ethnicity.html 
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Being at home made coursework really uninteresting because | could only listen to a lecture for 
20 minutes at a time, and then when | went back to it, | wasn't remembering what | heard from 
the last 20 minutes and all that. 


A student in a different biology class was appreciative that the instructor did not record full-length 
lectures but instead created a series of shorter videos: 


| thought [the professor] made a good choice for this class in breaking up the lectures. In the 
classroom setting when we had 50 minutes, we might’ve got between 30 and 40 minutes of 
actual lecture time. But rather than post a 35- or 40-minute lecture, they posted the same 
amount of lecture material but they broke them up into, like, eight videos. Each one was more of 
a bite-size... | have a hearing disability. In class | would’ve been one of the people interrupting 
[the professor] to make them repeat something. Once we were fully online, they wouldn’t have 
been able to get through the same amount of lecture material in the same amount of time. 
Having one long lecture wouldn’t have worked...So, | appreciated that choice that they made 
with the lectures. 


Admittedly, creating new course materials in smaller chunks placed a burden on instructors at a time 
when they too were impacted by the COVID pandemic. As one instructor explained, 


| found myself creating little video explanations for some of their [students’] questions...Going 
through those steps while you're watching the person was much more helpful than just being 
like, “Go here and click on there. | went through the whole thing step by step... It required a lot of 
work outside of class, on my end, to make sure that learning was still occurring. 


Active learning strategies 

The incorporation of active learning in a college course is one of the best-researched strategies for 
improving student learning (Freeman et al., 2014; Haak et al., 2011). The sudden shift to remote 
instruction in spring 2020 was not conducive to major changes in course design to make learning online 
more active, but many of the instructors already using active learning practices in their classroom-based 
courses did seek ways to preserve those elements after their courses went online. 


A practice reported by a number of the case study instructors was a guided or scaffolded note-taking 
activity that students were to complete while watching lecture videos. A biology instructor described the 
reasoning behind this practice. 


There was a lot of active learning, so they [students] would watch videos and turn in notes... 
Students love watching videos but they never extract the information from them. They think it 
just permeates to them, and so! just literally took the script and ask questions about the script 
so that they would extract the information that the video was trying to give them, and that was 
it. It was a very simple assignment but it went from passive to active. 


The geology instructor described earlier who broke each lecture into 2-3 shorter videos assigned a for- 
credit notetaking activity after the course went online. The instructor used this assignment to track 
students’ engagement with the course, 


The students were required to fill out a guided lecture outline that takes them through the notes in the 
presentation with me and they submit that on Bb [Blackboard] Learn as an assignment for their 
attendance...It’s basically an outline where the student can just follow along and fill in the blanks so they 
know how to organize their notes, they know what's important to write down, they know what's not 
important. 
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Focus group students in courses incorporating guided note-taking or other active learning activities 
during lectures did not report the same level of difficulty with motivation or staying engaged that 
other students did. |n most cases, required note-taking activities were a continuation of activities used 
in the course before COVID, which helped students maintain a routine of working steadily on the course 
after the transition to remote learning. As a biology student explained, 


Because of the way the class was structured already, it was pretty easy for all of that to stay the 
same because there were still days when you were supposed to post your notes. That meant you 
had to listen to all those video lectures and take notes on them, and then scan them and post 
them online by the due date. That kept that expectation pretty consistent. 


For another biology student, the note-taking activity was one of several assignments that were assigned 
with each weekly lecture. The student described the sum of these activities as being “task heavy” but 
acknowledged that they helped with learning the course material, 


In any given week you had, like, a video assignment that you had to review, take notes on, scan 
the notes, and then answer questions on it. Then, you'd have a concept map which was-- again, 
it was a document that you had to fill out with the concepts of that week, and then scan it and 
post that...You'd have to print notes to be ready for the lectures, because you had to take your 
written notes from the lectures on these specific pages. Then, you'd have to scan those and post 
them. You'd have a vocabulary quiz. Then, you'd have a weekly quiz as well...We did all of those 
different types of assignments every week. There was never a week that you didn't have that 
many of those. They hit every single one of those different types of learning outcomes every 
single week. Like | said, it ended up being somewhat task-heavy but at the same time, because 
you were reviewing all the materials in all those different ways, you couldn't help but learn it. 


Timely responses to student questions 

One of the aspects of in-person classes students reported missing the most was the ability to ask their 
instructor questions face-to-face and get an immediate response. Students missed being able to ask 
questions in real-time about their performance or their understanding of course topics. A math student 
who participated in focus groups and interviews explained, “I just miss being able to ask a question in 
that moment to elaborate on something.” A geology student shared, 


| miss being with people and asking the teacher questions directly. | think the hardest part about 
online for this course and every course was just spending five minutes after every class, talking to 
the teacher and asking random questions that came into my head during the class and emailing 
those questions feels way too formal. | don't want to waste my teacher's time emailing them. 


Students pointed out that the lack of immediate feedback was particularly deleterious to laboratory 
activities. 


There was not a good way to get help while working on any of the lab portions. [It was] Hard to 
know if you were doing things correctly with proper outcomes. 


Many instructors sought to mitigate loss of opportunities to ask questions in class by holding online 
office hours or review sessions with their students. A chemistry instructor, for example, described using 
Zoom for office hours and recitation help sessions. 


| had Zoom help sessions and personal office hours. Zoom is great, | can just set up my laptop in 
front of my whiteboard then it'd be like students sitting here in the office, and they could ask 
questions. | guess basically the big difference is the video's online using Zoom for office hours 
and help sessions. Attendance was about the same. The recitation was required. Everybody 
would come in for the recitation class on Zoom, but for office hours it was just as needed as they 
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[students] needed help. It's about the same as office hours, you now have maybe five students a 
week. 


A mathematics instructor who originally did not plan to hold live virtual office hours described their 
decision to do so. “At first, | thought I'd just use the videos that Wiley created,” said the instructor. 
“Then my students begged me to hold WebEx sessions, so | held a WebEx session every day during class 
time because they still wanted the interaction.” When asked what the most challenging aspect of 
remote learning was for their students, the instructor reflected, 


| think the ability to ask questions real-time, as they [students] were having the questions. 
Whether it was on homework problems or lecture notes, being able to ask those questions 
immediately, | think that was the hardest for them. That's why they wanted the WebEx sessions. 


Others instructors tried to be responsive to their students’ questions about course content by inviting 
students to email them with questions, 


| would respond to anyone who emailed me with concerns. | put announcements to everyone 
reminding them [students] and | posted in Canvas each week, a list of what they should do each 
day...| didn't tend to email students individually unless they had emailed me first. 


When you're lecturing [in person], even though they're not talking, you get a sense of what 
they're getting and what they don't get. They'll come up to you after the lecture and talk to you. 
You don't get any of that with the remote learning, especially the way that | did it with 
asynchronous. | told them that | was available and that they could email me and that we could 
web chat. 


Students generally appreciated having the opportunity to work with their instructor online in the ways 
described above, but some found it less satisfying than interacting face-to-face. One student, for 
example, described feeling uncomfortable asking questions during Zoom meetings in their math class. 


Well, being in class gives you the opportunity to ask questions right there. With the Zoom 
experience, | felt like | was interrupting the whole class. It wasn’t the same thing for her 
[instructor] to answer your questions right then and there like when we're in class. 


On the other hand, several instructors observed that some students who had not asked questions in 
face-to-face class sessions started doing so after the course moved online, 


In Zoom, | would ask a question and say, "Okay, everyone who thinks they have an answer, drop 
it into the chat box." | would see the chat box just start to fill up with answers. Some of them 
right, some of them wrong, they would prompt discussion. Students seemed a lot more willing to 
participate via chat than by speaking up out loud in class. | don't know if because they're at 
home behind a computer screen if that took away some of the embarrassment factor or the on- 
the-spot factor, but a lot of students were much more willing to leave comments or answers or 
responses in a Zoom chat. 


Another instructor noticed that some students liked to attend online office hours and simply lurk while 
others asked questions, 


Interestingly, | think a lot of students maybe felt more comfortable with the online office hours 
because they could just log on, mute their volume, not put on the video, and they could just 
listen, and they didn't have to physically go show their face anywhere or sit with their peers... 
Some of them are just shy and they maybe never would have come to office hours in person, but 
the fact that they can be completely anonymous and just be present and listen but not have to 
actually show their face or talk helps some of them to participate. 
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Students often sent questions to their instructors through email. Their satisfaction with this method 
depended on the instructor’s response time and response predictability. A biology lab student expressed 
frustration about the delayed response time getting answers to questions about lab assignments, 


There wasn't consistent communication all the time with the professor. There was just a lack of 
clarity of, "Okay, | have a question now, but when am | going to get this response?" It could be 
anytime between one or two days and the lab is due today. If | have a question about the lab, 
why even bother asking a question because I'm not going to get a response on time. 


Instructors, on the other hand, often felt worn down by student expectations to be available to answer 
questions 24/7, 


Yes, | would say | found myself working a lot more post-COVID than prior to COVID. There were a 
couple of students that expected responses to emails or to questions in the discussion forums 
within 10 minutes of them posting it. It'd be 5:30 at night and we would try to eat dinner and I’d 
be like, “I got to respond to this.” It was tough because there was no, “This is the end of the day 
work balance, and then okay, | can focus on my family.” That was no longer there. 


Using peer collaboration to promote understanding 

When students were asked what they missed about their pre-COVID course, many pointed to 
collaborating with peers. “I miss being able to talk to other people about the course,” said a biology 
student. “If | didn't understand something, | could turn to my friend sitting next to me and be like, ‘Oh, 
did you get that, do you understand that?’ ” Another student felt similarly about their mathematics 
course: “| miss asking questions and being able to ask my classmates if they understand something,” 
said the student. “If maybe | don't understand the teacher's way of teaching, maybe a student's way is 
easier for me to grasp the concept.” 


A student taking statistics shared a similar observation, 


| miss seeing my fellow peers. Working the problems out with the two other students that | was 
with helped me learn better and faster... There's someone else there instead of me just teaching 
myself or doing it [coursework] myself. 


Another mathematics student also valued peer support and noted that it was more difficult at a distance 
even for already formed study groups, 


| definitely miss the atmosphere of the classroom... The very first day of school, they [instructor] 
told us to go around the classroom and figure out who we're going to work with, and we made a 
group chat. We were very involved as classmates | would say. | definitely miss that because if | 
had a problem or if | didn't understand something, a friend was right next to me and they’d be 
able to help me. It was definitely more difficult to get in contact with those people once 
everything went down. 


In their interviews, course instructors were asked to describe the collaborative opportunities they 
provided for students, both before and after COVID. Few STEM instructors provided examples of 
including collaborative activities during remote learning, even when such activities were a part of the 
course before the coronavirus. For example, a biology instructor had a collaborative pre-COVID iClicker 
activity where students worked in pairs or small groups to answer questions. After the coronavirus 
outbreak, the iClicker activity was replaced with online weekly quizzes for which students submitted 
their answers individually. A student in a chemistry course that similarly replaced a peer learning activity 
with embedded quizzes after shifting to remote instruction explained the implications of losing 
opportunities to discuss course content with peers, 
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When | was in class the professor would put one [question] up on the projector and be like, 
“Okay, answer this question.” | would be able to talk to people around me and even discuss 
further in detail why the other answers are wrong but now since it's online, it kind of takes away 
the [discussion] element because after they [professor] present the question they say the answer 
and then we just submit our answers into a quiz in Canvas. It takes away the learning element of 
it, | guess. 


A mathematics instructor was one of the few faculty who described successfully transferring a 
collaborative learning experience into the online environment. Before COVID, the course included a 
recitation session that began immediately after lecture and was attended by the same students. 
Students formed small groups to work on assignments together during these sessions and if they had 
questions, they could ask the instructor or an undergraduate learning assistant, both of whom circulated 
the room. After COVID, the instructor continued the activity through a live synchronous session that 
made use of Zoom’s “breakout” groups. During breakout sessions, students worked on problems in 
small groups for approximately 12 minutes, after which they came back together as a larger group in 
Zoom to discuss the problem as a class. Students were then switched to a different breakout group to 
work on the next problem. 


More extended collaborative learning occurs when students engage in project-based learning, meeting 
outside of class time to work on a problem solution or joint product. On the national survey, 33% of 
students reported that their STEM course instructor had them work on group projects after COVID. 
Among the 28 STEM instructors who were interviewed about the case study courses, however, none 
described a course assignment or activity where students worked collaboratively outside of scheduled 
class or recitation time to produce a joint product or solution when the course was fully online. 


The mitigating role of prior use of adaptive courseware. 

Instructors who used adaptive courseware before COVID described spending less time than their 
colleagues did preparing their courses for the transition to remote instruction. As a math instructor 
explained, 


Because we're already so web-based, as far as our content goes, it wasn’t that long [to prepare 
the course to go online]. It was pretty easy to switch it over. | already had all their notes online. | 
already had Collaborate set up. | didn't have to do anything with my grading for that particular 
semester. | just had to put together a few documents and send out a few emails. It was really 
relatively easy for me. It was just, “Let's roll”... | would say that | spent maybe a few days, maybe 
eight hours. 


These instructors also reported that the transition to remote learning was fairly easy for their students. 
“In some ways, | think that we were lucky that we had a lot of online stuff set up already,” said a 
chemistry instructor. “As our schools transitioned, obviously, the lecture portion wasn't happening 
anymore, but a lot of their homework remained the same, so it was just substituting in those online 
lectures.” A geography instructor shared a similar comment about their liberal studies science course: 
“The students have used that [courseware] as their ‘textbook’ for the last two academic years. They 
were already fully doing all of their reading materials and their interactivity online before all this 
happened.” 


The mitigating role of prior use of adaptive courseware surfaced also in the interviews and focus groups 
with students. “I think for the most part they just kept it [course] consistent in terms of how they taught 
the class,” a biology student said. “They [professor] were able to just keep things going smoothly.” Fora 
chemistry student the prior use of adaptive courseware similarly eased the transition to remote learning 
and increased their course satisfaction: 
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This class is one of my classes that transitioned very well and has done a good job of providing 
lectures at the same time period...It was probably one of the smoother transitions because our 
homework was already online. | would say I'm very satisfied with this course. 


Instructor Responses to the COVID Circumstances: Navigating the Trade-offs 


STEM student and instructor reflections on their experiences with courses during COVID suggest that 
there is a trade-off between providing students with maximum flexibility to do their coursework 
whenever and wherever they could and having a predictable schedule to keep students involved in the 
course. Most instructors were given autonomy to design the post-COVID version of their courses as they 
saw fit, but a number of higher education institutions (including two of the nine in our case studies) 
either recommended or mandated that all remote instruction for courses that had previously held in- 
class meetings be done asynchronously. These institutions were concerned not only about students who 
might lack access to an adequate computing device and broadband internet connection but also about 
students who would no longer be in the same time zone as the instructor or who might have to work or 
take care of family members during the hours previously designated for class meetings. One chemistry 
instructor articulated this concern that live lectures would not be equitable for all students, “| did want 
to do synchronous classes,” said the instructor, “but then | recognized that not all students would be 
able to participate, and | didn't want to put the students that said they weren't able to attend in real 
time at a disadvantage.” In the end, this instructor did hold live synchronous lectures but recorded them 
for the benefit of students who could not attend during the regular scheduled lecture time. Building in 
options by offering lectures both in real time and as recordings posted online emerged as a common 
strategy at the case study institutions. As a biology instructor described, 


The first thing | did [after COVID] was record my lectures and deliver them asynchronously, and 
that was per our department mandate. They requested that we do that. After hearing from some 
of the students who might've struggled with the technology, | do think that at least recording 
them was a good idea, but | also heard from some students who said it was really hard for them 
to switch to online. 


Use of multiple delivery approaches within a course was common, as suggested by the percentage of 
survey respondents reporting use of synchronous and asynchronous delivery mechanisms in their STEM 
courses. (See Table 1.) 


Table 1. Delivery mechanisms in courses taught remotely after COVID 


Percentage of 
Students Reporting 


Delivery Element (n=620) 

Live sessions (synchronous instruction) 70% 
Recorded lectures by the instructor 

(asynchronous) 71% 
Pre-recorded videos, not of instructor 56% 


A number of instructors providing lectures during regular scheduled class times described their rationale 
that the format would encourage students to attend. A chemistry instructor reflected, “I feel like having 
the accountability of being there [lecture] at the time, kind of getting it live is definitely different than 
just watching the video secondhand.” According to a geology instructor, synchronous lectures were 
preferable because students would be likely to procrastinate watching recorded lectures. 
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| don't like asynchronous courses. It could just be a bias, but just like a goldfish will fill up to 
whatever size tank you put it in, a student will procrastinate to whatever deadline you give them. 
The students think, “If it’s asynchronous, well, then | can do it later.” We have to be as 
synchronous as possible. We still need to be asynchronous, but also offer a realistic avenue for 
those who can do it synchronously. 


Students themselves said that paying attention to online lectures, particularly those that were 
asynchronous, was challenging. Like students in the national survey sample, those participating in 
interviews or focus groups described missing “the reinforcement of having to show up to class.” One 
student described the struggle to stay on task at a distance. 


It was definitely a little bit hard to maintain the same level of focus [after COVID]. | was able to 
get through the lectures and | took notes and everything still, but at times it was pretty painful to 
do. It really took a lot of motivation to really sit through the full extent without fighting my mind 
wandering or whatever, and | didn't have that at all when we were in the course. | was fully 
engaged in it while we were in person. 


In student focus groups and interviews, we found that students taking courses with synchronous live 
class sessions did not report the same level of difficulty with procrastination or keeping to a schedule 
as those with entirely asynchronous courses did. Fewer than 5% of case study students with 
synchronous sessions brought up difficulties keeping up with the course compared to close to 25% of 
those in totally asynchronous courses. For the former, having their online lectures scheduled at a regular 
time each week contributed to their overall satisfaction with the course. A chemistry student whose 
course had synchronous recitation (tutorial) sessions explained how those sessions helped with the 
transition to remote learning. 


We met on Zoom and he [professor] said that we would meet every Tuesday like normal for 
recitation to answer questions. He would post the lectures at the start of each unit so there's 
four units, Monday, Wednesday, Friday. We’d watch about 15-minutes’ worth of his lectures 
because that's like a normal class time. If you do that, you'll be on pace for the test. 


The approach of having course lectures available both in real time and as recordings did not completely 
solve the tension between equity and motivation, however. Some students who had access to both 
synchronous and asynchronous options for viewing lectures described the latter as an invitation to 
procrastinate. 


The biggest challenge after COVID was just keeping up because when it was in-person, we would 
meet twice a week. But once the videos were posted online, then we could access them 
whenever we wanted, so it was really easy to push it off and say, “I'lldo that the next day,” and 
that was just kind of a cycle. 


A number of instructors developed strategies to deal with this procrastination issue by embedding 
quizzes or other required activities into the course materials that counted toward the student’s grade 
and had to be completed on a specified schedule. (See the descriptions under Active Learning Strategies 
above.) 


Others limited the time window within which instructional materials on a given course topic were 
available with the same goal of keeping students working steadily. 


We had an iClicker quiz that was supposed to be done every day. The idea was that you could 
then watch the lecture and then do the iClickers as a quiz instead of as something where you 
would talk to your neighbor and have it at that time. It really made it more of a mastery check 
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than it had been intended for. In terms of point structure, we kept everything worth the same 
amount. 


| had a mechanism for putting my lectures online, and so | felt confident that | could do that. | 
also developed a mechanism for them [students] to do participation. | recorded my lectures and 
posted them through the LMS and then the students would take notes and turn them in. 


Adapting assessment and grading practices to the circumstances of spring 2020 was another 
challenge for instructors. Most faculty were sensitive to the stressors on many of their students and 
wanted to avoid adding to those burdens through practices that would unfairly penalize students with 
limited technology access or competing responsibilities. On the other hand, STEM instructors value 
academic rigor and integrity and wanted their assessment and grading practices to motivate their 
students’ continued engagement and learning. Balancing compassion with standards that motivate hard 
work was a continuing challenge for instructors. 


Case study interviews revealed widespread adjustments to course assignments and due dates. A 
statistics instructor, for example, became more flexible with due dates after COVID. “A lot of my 
students have families and work and a lot of different responsibilities,” the instructor said. “I found 
myself being even more flexible with regards to deadlines. If a student requested to have an extension, | 
would be like, ‘Yes, sure. Go for it. Definitely.’ ” A mathematics instructor similarly described widespread 
relaxation of deadlines, “I would say I've had to reopen assignments that were not submitted by the 
deadline for two-thirds of the class at some point, and that was okay.” 


Other instructors introduced opportunities for students to receive extra points or credit after COVID. A 
chemistry instructor described an additional activity that students could complete to replace their 
lowest grade. 


We didn’t change the grading in the syllabus...but we wrote an extra review set for Connect 
[online homework] and we said, “If you do this, it'll replace your lowest grade,” because officially 
we weren't dropping any of those. 


Students appreciated their instructors’ flexibility around grading and deadlines. “I did like how he 
[professor] showed sympathy if we wanted to add an additional day for a test,” said a math student. 
“Whenever we would have a test, he would ask us if we wanted to open it for us for an extra day, and 
that was very helpful.” A biology student described how the deadlines for quizzes were extended, 


| definitely noticed that there were a lot of extensions allowed. Those quizzes that were supposed 
to be due on Fridays, | think my professor must have gotten bombarded by a lot of complaints 
about that deadline, because, | think, every single one of them got extended to a further date. | 
think she was trying to be flexible and willing to accommodate, which, I'm sure, a lot of students 
appreciated. 


Other students appreciated the opportunity to replace their lowest grade on an assignment or exam. In 
the words of a chemistry student, 


Originally, we had four exams and then one early exam that made up an enormous part of your 
grade. After the Coronavirus outbreak, she [professor] was going to drop our lowest exam score 
and just use our other three just to relieve some of that stress. What | actually was able to do is, 
because | did well enough on my first three exams, | didn't take my final exam because it would 
be dropped. 


Instructors struggled with how to assess learning meaningfully and fairly. Technology for proctoring 
tests given online was used widely during COVID, but not without controversy (Doffman, 2020). Some 
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institutions and faculty objected to the costs of the proctoring service. Others noted that the technology 
access differences that affected students’ learning also come to the fore when exams are given at a 
distance. In one chemistry department, two instructors decided to use Respondus, a lockdown browser 
that controls students’ testing environment within a learning management system. One of the 
instructors explained that students needed to have a webcam so they could be recorded through the 
lockdown browser. Those students who didn’t have a webcam were linked through Blackboard 
Collaborate to a teaching assistant who watched students on video through their phones as they took 
the exam. 


Some students expressed anxiety or resentment about having to use a lockdown browser system for 
taking their exams. 


I think my greatest challenge was motivation and using the ProctorU system. We were forced to 
use this proctoring service and after reading the privacy policy | did not agree with what | was 
signing up for but was told I'd fail the tests if | didn't. It's a really sketchy system and can see so 
much of your personal information on your screen | really think they could've found a much 
better alternative. | felt like my rights were taken away when | was forced to sign up for 
ProctorU. 


An alternative to lockdown browsers used by some instructors was the imposition of relatively tight 
windows for completing the exam. A chemistry instructor explained the rationale behind this approach. 


Students could take the exam unlimited times, but each time the student was limited to an hour. 
| didn't want them [students] to have it open for 24 hours and look everything up. They could 
have used their book, they could have Googled things, but | tried to use the time limit to squash 
how much they could look things up. 


Students with poor internet connections suffered in courses where quizzes and examinations were given 
with tight time windows. 


| definitely feel like it [internet crashes] stressed me out unnecessarily because we would have 30 
minutes to an hour for the quizzes on Friday, so you don't really have a lot of time to waste when 
you're working out math problems as hard as the ones we were doing. | never didn't finish, but it 
definitely triggered some anxiety. 


| just don't like the online format especially with the unstable internet that | have, because when 
| go into a test, | go in with more anxiety than | would normally if | were in person because | don't 
know whether my internet is going to go out or | have to contact someone to get it back, get the 
test reset, and if they're going to let me reset my test. 


Some instructors decided to change the nature of their assessments in light of the fact that they couldn’t 
prevent students from looking up information online or elsewhere during examinations. A mathematics 
instructor, for example, switched to using an open-book format for the final exam. 


Once we couldn't do [proctoring] anymore, we then needed to switch to online exams. | had 
static exams, meaning everybody got the same questions. | just told students, “Hey, you can 
have your notes available. You can use your book,” because | didn't feel it was reasonable to 
expect that they're not going to do that. 


Another mathematics instructor who also decided to use an open-book format did so primarily out of a 
sense of fairness. 


If anything, my expectations for the exam were a little higher since | told them... “It's an open 
book, open notes [exam]. Please don't use the internet.” I’m not going to tell them that. If you 
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tell the students they can't use their notes and they're sitting at home, the good ones that 
don’t...they're going to be penalized. Cheaters are going to cheat. | didn't feel comfortable with 
that. 


Students had mixed reactions to changes in their exam format. While some expressed nervousness or 
resentment about exam proctoring systems, there were others with unproctored exams who felt there 
was no longer a good reason to study. In the words of one chemistry student, 


Now we're having to take tests and quizzes and stuff at home, and they're not being proctored 
or anything. It's so easy to cheat or to go online, and you really don't have to study for any of 
your tests. Now, you're just not really learning much of anything. 


Hands-on laboratory experiences are an important part of many STEM courses. Both students and 
instructors expressed concern that remote instruction meant missing out on important lab work. 


I think my concern [with the course post-COVID] lies mostly with the labs because obviously labs 
are entirely hands-on and then when that was no longer an option. We're just watching people 
do all of these procedures, but more than likely, we're going to have to know how to do [them] in 
future biology programs. We didn't get to fill gel wells and do electro-freezes or dissect anything, 
which all seems pretty important for future classes, especially because you're just going to go 
further into those issues. 


| was doing research in microbiology with an assigned microbe for use in in vitro testing with wax 
worms. Due to leaving campus, | had to throw my bacterial cultures in the trash and "did" the 
lab virtually using a PowerPoint our professor put voice recordings over. We still had to write a 
research paper, but our professor had to make the rules more lax because we had no actual time 
in lab. 


One approach to dealing with the lab aspect of STEM courses was to have students view videos of the 
lab exercises being done by others, sometimes with instructor notations or requirements to answer 
some questions. 


Oh, my bio lab was not very good, | guess. | didn't really like it in general. It's said to be one of 
the harder labs for biology. We just watched this presentation video and filled out some 
questions on a document and submitted it every week. | didn't really feel like that was very 
effective learning. 


For labs, [what] he [instructor] did was he would find someone doing the lab on a YouTube video 
or something and he would post it, and we would watch it. He would give us important details 
like mass of a liter or something, and we had to find stuff like the volumes and stuff within it. 


Other instructors were able to have students do some lab work at home using common household items 
or online lab simulations. An anatomy instructor explained, 


A couple of the GAs [graduate assistants] reported that the reflexology [lab] still went pretty 
well. They just asked them to get a ruler from home, grab a partner, “Here this can work.” 
Instead of having the patellar hammer, the tap on the knee. You can get your own little hammer, 
just something to knock the front of the knee, and so, a couple of them reported that they 
[students] still enjoy doing the experiment at home. 


Several instructors reported having students work with virtual laboratories available in the course 
learning software. One student remarked, “My chem lab was an actual simulation and that was cool.” 
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Online lab exercises appeared to be more effective in students’ eyes when instructors were present 
online and used scaffolds and probes to direct attention to important aspects of the lab procedure. As 
one biology student explained, 


The instructions are still there. They ask questions throughout it that you can.... There's 
questions that they ask you on the online lab before you start the lab so that you understand the 
material better. Then, during the lab, they ask you questions to make sure if you're putting water 
in this, why are you putting water in this. If you're putting a cotton ball in between this acid and 
this, what are you doing, why are you doing it, what's the point. You understand the concepts 
better and how it relates to what you're doing. Then, they ask you questions at the end to see if 
you actually learned something. The format was the same and they also had ways of connecting 
the material that made it really easy for me to continue using it without struggling. 


Perceptions of Student Learning During COVID 


Only 3 of the 28 instructors we interviewed stated outright that they did not think students who took 
their course in spring 2020 would be prepared for the next course in the STEM sequence. Just over half 
of the case study course instructors gave positive replies when asked how well their spring 2020 
students were prepared for future STEM coursework (other instructors said they were not sure). A 
statistics instructor, for example, commented “If anything a think they were a little better, maybe, 
learned a little better.” A biology instructor gave a positive response that was a bit more guarded, 


The material in 181 [biology] seems to be recurrent material in bits and pieces throughout the 
career of a biology major, of a student majoring in biology, so | think they will have a chance to 
refresh. There is, | think, a solid foundation in the sense that a student will be able to go back and 
revisit and say, “Oh, yes, | remember this,” but | am not sure if they will remember right away. 


A number of instructors noted that their optimism that students had learned the course material 
remotely was confined to those students who continued participating in course activities after COVID. 


I try to structure the activities so that if you can't help but learn something when you're doing 
this and then that builds that encouragement and helps them. | think that there were a lot of 
tools and the ones that came prepared to learn and did the [assigned activities]—they learned 
something. 


The scores of those who stayed [in the course], they were fine. The test scores went up a lot, like | 
said, because tests were showing up wholesale online, but the participation dropped off. Those 
that dropped off mostly | just didn't hear from. 


Almost three-quarters of the students in interviews and focus groups said they thought they had still 
learned the course material despite having to finish their STEM course remotely. But that left more than 
a quarter of the students in focus groups and interviews who expressed concern that they had not 
learned as much as they would have if the course had continued with in-person classes. Students who 
were majoring in the subject of the STEM course they were describing were particularly likely to voice 
concerns about how well they would fare in their future STEM courses. 


This [remote learning] is kind of what's been making me reconsider being a biology student. It's 
just because learning online is so difficult. This is my 103. It's a 100-level biology class and 
everything that | learn after it is going to be based on the foundations that | learn in this class. 
I'm not learning these foundations very well, so it's like it's going to make everything after this 
harder because | didn’t learn any of this the way that | was supposed to or as well as | should 
have. 


STEM Courses During the COVID Pandemic: Lessons from Spring 2020 20 


I'm worried. My major is chemical engineering and | feel like this is probably the most important 
class to really have a mastery of because it is the foundation for everything, every other class I'm 
taking in the future. So, I’m definitely concerned that next year is going to be a struggle because 
I'm taking a whole bunch of kinds of chemistry classes and without fully understanding what I'm 
supposed to be learning right now, | definitely think it's going to be a big setback. 


I'm going to Organic Chem in the fall. Not having the basics, it's going to hurt. I'm going to have 
to devote way more time to chemistry now than I'm pretty sure the typical person would have to 
because I'm going to have all these little spaces missing at the end of 105, which is basically the 
foundation for the rest. 


I'm actually going to be taking a business calculus class in the fall and I'm really nervous that it's 
going to be pushed online...I'm a little nervous because | know that that class takes a lot more 
effort and | want to do good and | want to learn it. I'm just afraid that of it going online because 
the teacher didn’t seem like he would be too tech-savvy, so it would probably just add to the 
problem. 


It [remote learning experience] definitely discourages me from taking any kind of bio classes 
online again because it's a nightmare. | think that | definitely didn't get as much out of the bio 
lab class as someone would've gotten if they took it the fall before all of this happened where it 
was just normal. | don't think | learned as much or understood the concepts as much as someone 
who took the class before me, which makes me feel at a disadvantage if that material comes up 
later on. If we have to understand a concept from earlier, | won't know it as well. 


I'm scheduled to take organic chemistry in the fall, which is known to be one of the hardest 
chemistry [courses] at Ole Miss. Part of me is super nervous that we're going to need the skills 
that we were supposed to learn in this class...I'm really nervous that we're supposed to be able to 
understand concepts from this semester, and we're just not going to be able to because we 
weren't really taught. We were just shoved into figuring out how to do it for just for the test, and 
when we need them next year, we're just not going to have the basic skills to do the work. 


Instructors’ Use of Recommended Practices for Online Instruction 


We have described the nature of STEM courses as they were conducted remotely in response to COVID 


as well as some of the tradeoffs and concerns faculty dealt with in deciding how to run their courses. 


While many press accounts of remote instruction during COVID decried its quality (Hubler, 2020), 


proponents of online learning made the point that what was occurring in spring 2020 was emergency 


remote instruction rather than online learning, as it is generally defined and practiced in the field 
(Gallagher & Palmer, 2020; Hodges et al., 2020). 


Designing and implementing an effective online course--or any course for that matter--can be 
conceptualized as attending to three essential “Cs:” Cognitive, Community, and Communication. 


e The Cognitive dimension of a course is the one most faculty are accustomed to thinking about. It 


includes identification of the key learning outcomes for the course, design of meaningful 
assessments of those outcomes, strategies for encouraging active (“minds on”) learning. 


e The Community dimension of a course involves building a learning community in which all 


students feel they belong (“rightful presence”) and are supported. Use of real-world examples 


touching on students’ lives can draw on their prior knowledge and promote a sense of 


togetherness. Instructor responsiveness to student questions and needs and the instructor’s 
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demonstration of respect for all students help build community as do opportunities to learn 
with fellow students through peer learning, study groups, and team projects. 


e Finally, instructors should have a plan for the Communication dimension of their course. This 
encompasses being very clear about course expectations and standards for performance, 
providing information on how and where students can get help with course content and can 
access other kinds of support, and personal personalized suggestions to individual students for 
how to further their learning. 


Survey Data on Recommended Practices 
We realized that spring 2020 did not afford the time for colleges and instructors to carefully design and 
implement an online course with all of the features of best practice. Nevertheless, we wanted to 
understand the nature of remote instruction during COVID vis a vis that ideal. We included items on the 
national student survey asking students whether their course after COVID included each of eight 
instructional practices chosen because they could be expressed in concrete everyday language ona 
survey for students and are prominent in the higher education research and practitioner literatures 
(American College and University Educators, n.d.; Darby, 2019; Luskin, 2019; Orlov et al., 2020; Salcido & 
Cole, 2018; Siemens, Gasevic, & Dawson, 2015) on effective teaching and learning online. Specifically, 
the eight practices representing the three Cs of quality instruction were: 


e Assignments that ask students to express what they have learned and what they still need to 
learn (cognitive) 

e Breaking up class activities into shorter pieces than in an in-person course (cognitive) 

e Frequent quizzes or other assessments (cognitive) 

e Live sessions in which students can ask questions and participate in discussions (communication) 


e Personal messages to individual students about how they are doing in the course or to make 
sure they can access course materials (communication) 


e Meeting in breakout groups during a live class (community) 
e Using real-world examples to illustrate course content (community) 


e Work on group projects separately from the course meetings (community) 


Figure 2 shows the proportion of students in STEM courses reporting that their course included each of 
these features when it was taught remotely. For comparison purposes, survey responses for non-STEM 
courses are shown as well. 


The prevalence of these recommended practices during COVID instruction varied. A majority of students 
describing their STEM courses reported that they included live sessions during which they could ask 
questions, frequent quizzes or other assessments, and use of real-world examples to illustrate course 
concepts. Half of the STEM students reported that their instructors sent personal messages to them, and 
almost half reported having assignments requiring them to reflect on what they had learned and what 
they still needed to learn. Other recommended practices—meeting in breakout groups during a live 
class, breaking up class activities into shorter segments, and group projects outside of class time—were 
used in a third of less of the STEM courses. Compared to STEM courses, instructors of courses in non- 
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Figure 2. Proportion of survey students reporting recommended instructional practices in their 
STEM courses 
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STEM subjects were more likely to send personal messages to individual students, break class activities 
up into shorter segments, and to ask students to reflect on what they had learned and what they still 
needed to learn. STEM courses, on the other hand, were more likely to employ frequent quizzes or 
other assignments. 


Another item on the Survey of Student Perceptions of Remote Teaching and Learning asked students to 
rate their satisfaction with their STEM course after it went to remote delivery on a 4-point scale (very 
satisfied, somewhat satisfied, somewhat dissatisfied, very dissatisfied). Analysis of differences in course 
satisfaction responses for courses with and without each of the recommended instructional practices 
revealed that those practices most strongly related to students’ course satisfaction were 


e Personal messages from the instructor on how the student was doing or to make sure they 
could access course materials. Sixty-four percent of students who received messages of this type 
were Satisfied with their STEM course overall, compared to 50% of those who did not. 

e The instructor's use of real-world examples to illustrate course content. Among students whose 
instructor did this in the online course, 65% were satisfied overall, compared to a 42% 
satisfaction level among those whose instructor did not use real-world examples. 

e Assignments requiring students to express what they had learned and what they still needed to 
learn. Satisfaction with online STEM courses that included such assignments was 67% compared 
to 49% for courses without this feature. 


We note that two of these influential practices were ones that were more prevalent in non-STEM 
courses than in STEM courses. These findings have implications for encouraging STEM instructors to 
participate in professional learning around online teaching practices. 
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We also investigated the influence of including multiple recommended practices in the course after 
COVID. The number of the eight recommended practices that students said their course included was 
tabulated for each survey respondent. Figure 3 shows that the number of these online instructional 
practices incorporated in the course predicted students’ satisfaction with their STEM course after 
COVID. As seen in the top bar in the figure, 75% of students in STEM courses using 6 or more of the 
recommended instructional practices were either somewhat or very satisfied with their course as it 
was taught during COVID. 


Figure 3. Satisfaction with post-COVID course by number of recommended online practices used 
in instruction 
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Among survey respondents, staying motivated to do well in the course emerged as the most widespread 
barrier for students after their courses moved online due to COVID. The routines of attending class in 
person had provided a way for students to hold themselves “accountable,” and some found that they 
were not prepared for the level of learner self-direction required in an online format, especially when 
classes were held asynchronously. Disruptions associated with work and family life further exacerbated 
the demotivating effects of losing structure and personal interaction. Additionally, emotional distress 
and uncertainty caused by the pandemic undermined students’ attempts to focus on academic work. 
Students from minoritized groups and those from lower-income households were differentially 
impacted by challenges other than motivation. Survey and case study also call attention to the 
importance of college campuses as physical spaces for student success. Campuses provide students with 
separation from personal and work life and thus a place to work without daily distractions. They also 
provide facilities upon which students rely, especially WiFi and access to computing devices and 
laboratories. 


Reduced Opportunities for Active and Collaborative Learning 


Active learning activities calling on students to think deeply about course concepts (Freeman et al., 
2014; Haak et al., 2011; Prince, 2004) and social construction of knowledge (Dabbagh et al., 2019; 
Vygostsky, 1978) are two of the most effective instructional practices in the research literature. A 
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number of specific techniques, such as peer-assisted learning and project-based learning, combine these 
two concepts to engage students in deeper learning (Lewis & Lewis, 2005). Many instructors lacked the 
tools and preparation time needed to design online course activities involving student collaboration and 
active learning. Only 3 of the 28 case study course instructors we interviewed described creating 
breakout groups in Zoom to allow for small-group discussion. None of the instructors interviewed for 
STEM course case studies described assigning group projects after COVID. Even a case study professor 
who typically used a highly interactive, evidence-based pedagogical strategy (process-oriented guided 
inquiry learning or POGIL) for his general chemistry course did not know how to translate this approach 
into an online environment. Thus, while instructors were justifiably wary of placing too many demands 
on students unaccustomed to online course environments, the fact that so many remote STEM courses 
lacked opportunities for even informal group problem solving during class time may have added to 
students’ feelings of isolation. 


The net result of the COVID-driven transition to remote instruction was that students were somewhat 
less likely to experience instructional practices designed to elicit active learning, and much less likely to 
have opportunities for social construction of knowledge. Many courses, by necessity, reverted to more 
traditional, didactic methods relying upon video lectures and traditional assessments (e.g., multiple- 
choice quizzes and tests rather than performance assessments or complex problem-solving requiring 
transfer of knowledge). These changes in practice may have exacerbated academic challenges for low- 
income and minoritized students, who have been found to benefit disproportionally from active 
learning pedagogies in STEM disciplines (Freeman et al., 2014; Prince, 2004). 


Balancing Equity and Rigor 


Instructors interviewed for the course case studies expressed empathy for their students and described 
ways in which they had relaxed deadlines and some of their normal course requirements in light of the 
challenges some of their students were facing. Instructors were motivated by the desire to reduce the 
anxiety their students were feeling and be fair to students who did not have the technology access or 
quiet space and dedicated time for learning that they had had before COVID. Most students appreciated 
these instructor demonstrations of empathy and flexibility, but a downside to these instructor responses 
emerged as well. Without deadlines, some students let their course work slip and had a hard time 
recovering. Similar issues and trade-offs emerged for course examinations. Concern with cheating led 
some instructors (and entire campuses) to require proctoring software, a practice that some students 
found intrusive and anxiety provoking. Use of tight time windows to discourage students from Googling 
the answers to test items was another practice that raised student anxiety, especially for those with 
poor internet connections or disruptive activities within their homes. Open-book examinations without 
time limits, on the other hand, were sometimes perceived by students as requiring less preparation and 
raised student concerns that their peers were cheating. 


Consistency Within and Across Courses 


Another theme found in student comments concerned the benefits of consistency (and confusion 
caused by inconsistency) in course expectations before and after the shift to remote instruction. Student 
and instructor interviews suggested that use of adaptive and interactive courseware systems smoothed 
the transition online because this component of courses was unchanged and provided structure. 
Students in courses featuring adaptive courseware were already accustomed to doing coursework 
online and knew how to seek help within the systems (for those that provided intelligent tutoring). In 
other words, the learning curve necessitated by COVID was less steep for both instructors and students 
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in courses that were already using online learning systems as part of their STEM courses because the 
remote version of the course was largely consistent with the version taught pre-COVID. 


The toll taken by lack of consistency across courses was highlighted by a number of students who 
pointed to the fact that the different courses they were taking used different approaches, routines, and 
technologies. This occurred because instructors, even within the same department, typically designed 
their courses in isolation, making different decisions about how lectures were delivered, whether to 
hold online office hours, and incorporation of mechanisms for active learning and assessment online. 
Some instructors said they would have appreciated more prescriptive guidance from their departments 
about how to structure their online courses and which technology tools to use. 


There's a lack of coordination among faculty and even though there’s e-learning professional 
development, there's just a severe lack of coordination among faculty at NAU in terms of delivery 
of content for our students. It didn't surprise me that the administration was like, “Just do it.” 
and everyone just designed their courses in their own way as they saw fit, without a lot of 
guidance and without a lot of pedagogical background into what they're doing. 


Some instructors also pointed out that institutional laissez faire policy on how to handle the transition to 
remote instruction placed a burden on students to employ many different technology tools and keep 
track of instructors’ different course structures and requirements. As one instructor commented, 


| would have appreciated if our department as a whole said—maybe even the whole school— 
would've said: “This is how we're transitioning. You are required, as a faculty member, to meet 
with your students and go through the lecture or you're required to have office hours and pre- 
recorded videos.” Because | feel the students when they were coming back, they were just like, 
“What am | even doing?” They were receiving all these e-mails, and they were getting confused 
between, well, “In this class I'd have to do the lectures and | have to watch them beforehand, but 
in this class, | have to a different task.” | just think the students had a lot of confusion as to what 
the classroom was going to be. 


Recommendations 


As the COVID-19 pandemic continues and many students continue to be faced with the choice of taking 
courses online or not taking them at all, the spring 2020 experiences of students and instructors in STEM 
courses suggest steps that colleges and instructors should take. 


For Higher Education Institutions 


At the institutional level, as long as there is a possibility that students will have to take their courses 
entirely or partially remotely, we recommend: 


e Ensuring every student has a computer adequate for videoconferencing, the learning 
management system, and any digital learning system used in the college’s courses. Some 
colleges are providing technology to students who need it or making the technology a basic 
requirement so that it can be covered by student scholarship and loan money. 


e Providing both synchronous and asynchronous learning options within each course to provide 
the flexibility some students need without sacrificing the sense of instructor presence and 
support most students find motivating. 
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Using institutional resources to communicate with students through multiple channels and to 
ensure continuation of support and resources such as tutoring and supplemental instruction if 
the campus has to close. 


Establishing expectations at the institution or department level about online course structure to 
ensure greater consistency in student experience across courses in the event that instruction 
must be done at a distance. 


Providing support for faculty members to learn how to incorporate evidence-based pedagogical 
practices into online instruction. In particular faculty need support in developing online analogs 
for lab activities, using tools that enable student collaboration, and designing assessments that 
emphasize the application of course content rather than memorizing factual content that can be 
looked up online. 


For large, introductory STEM courses, staffing the course with a second instructor or teaching 
assistants is desirable in light of the combination of teaching and technology challenges in 
delivering high-quality instruction online. 


At the same time, institutions and departments need to be thinking about how students who took their 
STEM courses during spring 2020 will be served when taking future courses in their STEM sequence. 
Student reflections on their preparation future STEM courses and instructor reports of reductions in 
course coverage and relaxed grading mechanisms suggest that the spring 2020 COVID learning 
experiences may have long-term consequences for students if additional supports are not offered. We 
recommend that departments work with faculty teaching the STEM courses that students take after 
their introductory course to: 


develop plans for dealing with broader-than-usual ranges of prior knowledge and competencies 
among students enrolling in the course. Implementing strategies such as supplemental 
instruction and adaptive learning courseware can help students who took their introductory 
STEM courses during the COVID pandemic catch up on what they may have missed or failed to 
master. 


hone techniques for expressing empathy with the conditions that students have struggled with 
in doing their STEM learning online. 


For Instructors and Course Designers 


For course instructors and those who help them design their courses, we offer two overarching 
recommendations. 


First, consider incorporating independent digital learning resources and activities into STEM 
courses even if face-to-face instruction resumes. These activities provide opportunities for self- 
regulated learning that prepare students for future learning and can be adapted to individual 
student needs. In the event that circumstances change and students have to do some of their 
learning at a distance, they will be better prepared to continue their coursework and the 
transition to online instruction will be less disruptive. 


Second, we encourage those designing college courses to attend to the course’s three Cs-- 
Cognitive, Community, and Communication—regardless of whether the course will be taught 
online, face-to-face, or in a blended mode. Because it is always possible that a course will have 
to be taught online, all faculty should be familiar with techniques for fostering cognitive 
engagement, community, and communication online. In the Cognitive realm, areas we view as 
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in need of greater attention include (1) techniques for fostering active learning online, 
especially in courses with large enrollments, and (2) meaningful, “Google-proof” learning 
assessments that call for the application of knowledge in solving problems. In the Community 
realm, we urge course designers to develop a community-building strategy for their courses. 
Especially in times of crisis such as the COVID pandemic, students’ emotional connection to their 
instructor and peers and their feeling of belonging in the course are important in motivating 
them to learn. To the extent the STEM courses in our case studies are representative, STEM 
courses appear to make little use of techniques that have students work together. Peer learning 
in breakout groups and techniques for establishing an online “presence” can enhance students’ 
sense of belonging to a course community even when learning must be done at a distance. 
Finally, regular communication with students should be regarded as an essential teaching 
function. Instructors should develop communications plans that include how they will 
repeatedly inform students about how to access available resources providing help with the 
course and other kinds of student support, provide clear statements and illustrations of criteria 
for success, and specifications for when and how they will personally reach out to students who 
are struggling or not participating. 


The changes in instructional practices that we recommend can be supported by both general-purpose 
and domain-specific digital learning tools, which are increasingly available for STEM subjects such as 
mathematics, statistics, anatomy and physiology, biology, chemistry, and physics. Simulations and 
virtual reality technology can support some STEM lab activities at a distance. Adaptive courseware offers 
diagnostic assessments of student competencies that can be used to develop individualized learning 
plans for each student, helping faculty deal with the variability in students’ prior preparation. Reflection 
prompts for students to write about aspects of course material they have reviewed but don’t feel they 
understand are built into a number of courseware products. Learning management systems have 
features that support student-instructor and student-student communications. Some learning systems 
automatically generate lists of students who have not logged in for a specified time period or who have 
fallen significantly behind the expected rate for mastering course objectives. These systems help faculty 
communicate with students by generating template messages that faculty can send as is or edit as they 
choose. We recommend ramping up faculty learning about the capabilities of digital learning tools and 
how best to implement them. 


Whether or not it involves digital learning tools, course redesign and transformational changes in 
instructional practice require a significant investment of time and effort coordinated at the institutional 
or at least the departmental level. The course improvement process will not be fast or easy, but serving 
students equitably and well demands it. 
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